[Feature][history server] support endpoint /events by seanlaii · Pull Request #4479 · ray-project/kuberay

seanlaii · 2026-02-04T11:21:55Z

Why are these changes needed?

The Ray Dashboard provides an /events endpoint that returns cluster-level events for observability. When a Ray cluster is deleted, this data is lost. This PR implements the /events API for the History Server, enabling users to query historical RayEvents from deleted clusters with a response format compatible with the Ray Dashboard.

Background: Two Event Systems in Ray

Ray has two distinct event systems:

Dashboard Cluster Events (event.proto) - Human-readable messages stored in logs/events/
RayEvents / Export Events (events_base_event.proto) - Structured task/actor lifecycle data exported via the Export API

The History Server Collector stores RayEvents (Export Events), which contain rich task/actor data. This implementation transforms RayEvents into a Dashboard-compatible response format.

Summary

This PR adds:

Event types (Event, EventMap, ClusterEventMap) in types/event.go following the existing Task/Actor pattern
RayEvent transformation (transformToEvent, extractHostnameAndPid, extractJobIDFromEvent) in eventserver.go
/events handler in router.go supporting both live cluster proxy and historical data retrieval
Unit tests for Event types with comprehensive coverage
E2E tests for both live and dead cluster scenarios

Design Decisions

Max 10,000 events per job: Matches Ray Dashboard's MAX_EVENTS_TO_CACHE constant
FIFO eviction policy: Drop oldest events when limit exceeded, keeping newest data
Group by jobId or global: Node/lifecycle events without jobId are stored under global key
camelCase JSON field names: Match Ray Dashboard format for frontend compatibility
Use types.EventType constants: Type-safe event type references instead of raw strings

API Response Format

{
  "result": true,
  "msg": "All events fetched.",
  "data": {
    "events": {
      "<jobId>": [
        {
          "eventId": "...",
          "eventType": "TASK_DEFINITION_EVENT",
          "sourceType": "GCS",
          "timestamp": "2026-01-16T19:16:15.210327633Z",
          "severity": "INFO",
          "label": "TASK_DEFINITION_EVENT",
          "nodeId": "...",
          "sourceHostname": "ray-head-0",
          "sourcePid": 12345,
          "customFields": { ... }
        }
      ],
      "global": [ ... ]
    }
  }
}

Fields with Partial Availability

Unlike Dashboard Cluster Events which always populate sourceHostname and sourcePid, RayEvents only contain this information in specific nested event types:

sourceHostname: Extracted from NodeDefinitionEvent.hostname, available in NODE_DEFINITION_EVENT only
sourcePid: Extracted from workerPid, pid, or driverPid, available in TASK_LIFECYCLE_EVENT, ACTOR_LIFECYCLE_EVENT, DRIVER_JOB_DEFINITION_EVENT
message: Usually empty in RayEvents (data is stored in nested event fields instead)

NOTE: This is a fundamental difference between the two event systems in Ray. Dashboard Cluster Events are designed for human-readable logging, while RayEvents are structured for programmatic analysis.
Known Limitation: Event deduplication is not implemented. Duplicate events may occur if the same files are re-read (e.g., hourly refresh). This will be addressed in a follow-up issue along with the existing file tracking TODO.

Related issue number

Closes #4380

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

Signed-off-by: seanlaii <qazwsx0939059006@gmail.com>

historyserver/pkg/eventserver/eventserver.go

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

cursor · 2026-02-05T15:54:32Z

historyserver/pkg/eventserver/eventserver.go

+
+			// Extract sourceHostname and sourcePid from nested events where available
+			extractHostnameAndPid(event, eventType, data)
+		}


Task profile event data may be silently lost

Medium Severity

The field taskProfileEvents is uniquely plural among all event data fields (others are singular like taskDefinitionEvent, actorLifecycleEvent). This naming suggests it may contain an array of profile events rather than a single map. The code only handles map[string]any type assertions - if the actual structure is []any, the type assertion silently fails, causing task profile data to not be captured in customFields and jobId extraction to fail (defaulting to "global" grouping).

Additional Locations (1)

historyserver/pkg/eventserver/eventserver.go#L304-L313

The concern is not valid. The taskProfileEvents field is a single object (map), not an array. Here's the evidence:

Proto Definition:

message TaskProfileEvents { bytes task_id = 1; int32 attempt_number = 2; bytes job_id = 3; ray.rpc.ProfileEvents profile_events = 4; // the "events" array is INSIDE here }

Actual JSON Data

{ "eventType": "TASK_PROFILE_EVENT", "taskProfileEvents": { "attemptNumber": 0, "jobId": "BAAAAA==", "taskId": "...", "profileEvents": { "componentId": "...", "componentType": "worker", "events": [...] // <-- the array is HERE, inside profileEvents } } }

seanlaii · 2026-02-07T01:33:12Z

Hi @CheyuWu @troychiu @justinyeh1995 , please help review the PR when you have a chance. Thank you!

justinyeh1995 · 2026-02-07T05:32:54Z

historyserver/pkg/historyserver/router.go

+	var response map[string]any
+
+	if jobID != "" {
+		events := s.eventHandler.ClusterEventMap.GetByJobID(clusterSessionKey, jobID)


just want to double-check the behavior here, /events?job_id= would also return all events. Is that intended?

No, it should only return the events related to the specified job.

seanlaii added 2 commits February 4, 2026 05:38

[Feature][history server] support endpoint /events

df7774d

Signed-off-by: seanlaii <qazwsx0939059006@gmail.com>

use constant type

ba93876

Future-Outlier self-assigned this Feb 4, 2026

seanlaii added 2 commits February 4, 2026 13:12

change key name

b00e39d

convert timestamp

7ddb657

seanlaii marked this pull request as ready for review February 4, 2026 19:39

cursor bot reviewed Feb 4, 2026

View reviewed changes

historyserver/pkg/eventserver/eventserver.go Show resolved Hide resolved

Copilot AI mentioned this pull request Feb 5, 2026

Review all open pull requests #4482

Closed

4 tasks

cursor bot reviewed Feb 5, 2026

View reviewed changes

historyserver/pkg/eventserver/eventserver.go Outdated Show resolved Hide resolved

address timestamp comment

2e2bd26

seanlaii force-pushed the event-api branch from 6c845b6 to 2e2bd26 Compare February 5, 2026 15:42

cursor bot reviewed Feb 5, 2026

View reviewed changes

justinyeh1995 reviewed Feb 7, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature][history server] support endpoint /events#4479

[Feature][history server] support endpoint /events#4479
seanlaii wants to merge 5 commits intoray-project:masterfrom
seanlaii:event-api

seanlaii commented Feb 4, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Uh oh!

cursor bot Feb 5, 2026

Uh oh!

seanlaii Feb 5, 2026

Uh oh!

seanlaii commented Feb 7, 2026

Uh oh!

justinyeh1995 Feb 7, 2026 •

edited

Loading

Uh oh!

seanlaii Feb 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

seanlaii commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Background: Two Event Systems in Ray

Summary

Design Decisions

API Response Format

Fields with Partial Availability

Related issue number

Checks

Uh oh!

Uh oh!

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor bot Feb 5, 2026

Choose a reason for hiding this comment

Task profile event data may be silently lost

Uh oh!

seanlaii Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

seanlaii commented Feb 7, 2026

Uh oh!

justinyeh1995 Feb 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seanlaii Feb 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

seanlaii commented Feb 4, 2026 •

edited

Loading

justinyeh1995 Feb 7, 2026 •

edited

Loading